An image processing algorithm typically calculates a figure of merit for a series of real images, and the image having the largest figure of merit is assumed to be the “best” image. Reference in this regard may be made to, for example, F. C. A. Groen, I. T. Young, G. Lighthart, “A comparison of different focus functions for use in autofocus algorithms”, Cytometry, Vol. 6, pgs. 81-91 (1985). The quality and effectiveness of the algorithm is typically tested by comparing the image selected using the algorithm with an image that a human selects as the “best” image. Reference in this regard may be made to, for example, any of the following publications (in addition to the Groen et al. publication noted above): A. Santos, et. al., “Evaluation of autofocus functions in molecular cytogenic analysis”, J. Microscopy, Vol 188 (3), pp 264-72, (1997); J. M. Geusebroek, F. Cornelissen, A. Smeulders, H. Geerts, “Robust Autofocusing in Microscopy”, Cytometry, Vol. 39, pgs. 1-9 (2000); Y. Sun, S. Duthaler, and B. J. Nelson, “Autofocusing in computer microscopy—selecting the optimal focus algorithm,” Microscopy Research and Technique, Vol. 65, No. 3, pgs. 139-149, 2004; Y. Sun, S. Duthaler, and B. J. Nelson, “Autofocusing algorithm selection in computer microscopy,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2005), Edmonton, Alberta, Canada, Aug. 2-6, 2005; and X. Y. Liu, W. H. Wang, Y. Sun, “Dynamic evaluation of autofocusing for automated microscopic analysis of blood smear and pap smear”, J. Microscopy Vol. 227(1), pgs. 15-23 (2007).
In a conventional approach, shown in FIG. 1A, a first step applies an image processing algorithm to a series of real images to obtain a figure of merit for each image. A next step then uses a human to rank order the images by perceived quality. A determination is then made to determine if the rank ordering by human judgment agrees with the order based on the algorithmic figure of merit. If it does, the result is inconclusive since both the algorithm and the human judgment may be either correct or incorrect. If the rank ordering based on human judgement is found not to agree with the algorithmic figure of merit the result is also inconclusive, since either the algorithm or the human judgment may be correct.
As may be appreciated, this conventional approach is subjective and error prone. In addition to the variability of human judgment, for some imaging situations neither the algorithm nor the human actually select the “best” image. Thus, conventional methods of evaluating image processing algorithms, as outlined in FIG. 1A, are inadequate and error prone.